SEO company Parramatta

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually <meta name="robots" content="noindex"> ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish to crawl. Pages typically prevented from being crawled include login-specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.^[43]

In 2020, Google sunsetted the standard (and open-sourced their code) and now treats it as a hint rather than a directive. To adequately ensure that pages are not indexed, a page-level robot's meta tag should be included.^[44]

A variety of methods can increase the prominence of a webpage within the search results. Cross linking between pages of the same website to provide more links to important pages may improve its visibility. Page design makes users trust a site and want to stay once they find it. When people bounce off a site, it counts against the site and affects its credibility.^[45]

Writing content that includes frequently searched keyword phrases so as to be relevant to a wide variety of search queries will tend to increase traffic. Updating content so as to keep search engines crawling back frequently can give additional weight to a site. Adding relevant keywords to a web page's metadata, including the title tag and meta description, will tend to improve the relevancy of a site's search listings, thus increasing traffic. URL canonicalization of web pages accessible via multiple URLs, using the canonical link element^[46] or via 301 redirects can help make sure links to different versions of the URL all count towards the page's link popularity score. These are known as incoming links, which point to the URL and can count towards the page link's popularity score, impacting the credibility of a website.^[45]

SEO techniques can be classified into two broad categories: techniques that search engine companies recommend as part of good design ("white hat"), and those techniques of which search engines do not approve ("black hat"). Search engines attempt to minimize the effect of the latter, among them spamdexing. Industry commentators have classified these methods and the practitioners who employ them as either white hat SEO or black hat SEO.^[47] White hats tend to produce results that last a long time, whereas black hats anticipate that their sites may eventually be banned either temporarily or permanently once the search engines discover what they are doing.^[48]

An SEO technique is considered a white hat if it conforms to the search engines' guidelines and involves no deception. As the search engine guidelines^[11][12][49] are not written as a series of rules or commandments, this is an important distinction to note. White hat SEO is not just about following guidelines but is about ensuring that the content a search engine indexes and subsequently ranks is the same content a user will see. White hat advice is generally summed up as creating content for users, not for search engines, and then making that content easily accessible to the online "spider" algorithms, rather than attempting to trick the algorithm from its intended purpose. White hat SEO is in many ways similar to web development that promotes accessibility,^[50] although the two are not identical.

Black hat SEO attempts to improve rankings in ways that are disapproved of by the search engines or involve deception. One black hat technique uses hidden text, either as text colored similar to the background, in an invisible div, or positioned off-screen. Another method gives a different page depending on whether the page is being requested by a human visitor or a search engine, a technique known as cloaking. Another category sometimes used is grey hat SEO. This is in between the black hat and white hat approaches, where the methods employed avoid the site being penalized but do not act in producing the best content for users. Grey hat SEO is entirely focused on improving search engine rankings.

Search engines may penalize sites they discover using black or grey hat methods, either by reducing their rankings or eliminating their listings from their databases altogether. Such penalties can be applied either automatically by the search engines' algorithms or by a manual site review. One example was the February 2006 Google removal of both BMW Germany and Ricoh Germany for the use of deceptive practices.^[51] Both companies subsequently apologized, fixed the offending pages, and were restored to Google's search engine results page.^[52]

Companies that employ black hat techniques or other spammy tactics can get their client websites banned from the search results. In 2005, the Wall Street Journal reported on a company, Traffic Power, which allegedly used high-risk techniques and failed to disclose those risks to its clients.^[53] Wired magazine reported that the same company sued blogger and SEO Aaron Wall for writing about the ban.^[54] Google's Matt Cutts later confirmed that Google had banned Traffic Power and some of its clients.^[55]

SEO is not an appropriate strategy for every website, and other Internet marketing strategies can be more effective, such as paid advertising through pay-per-click (PPC) campaigns, depending on the site operator's goals.^{[editorializing]} Search engine marketing (SEM) is the practice of designing, running, and optimizing search engine ad campaigns. Its difference from SEO is most simply depicted as the difference between paid and unpaid priority ranking in search results. SEM focuses on prominence more so than relevance; website developers should regard SEM with the utmost importance with consideration to visibility as most navigate to the primary listings of their search.^[56] A successful Internet marketing campaign may also depend upon building high-quality web pages to engage and persuade internet users, setting up analytics programs to enable site owners to measure results, and improving a site's conversion rate.^[57][58] In November 2015, Google released a full 160-page version of its Search Quality Rating Guidelines to the public,^[59] which revealed a shift in their focus towards "usefulness" and mobile local search. In recent years the mobile market has exploded, overtaking the use of desktops, as shown in by StatCounter in October 2016, where they analyzed 2.5 million websites and found that 51.3% of the pages were loaded by a mobile device.^[60] Google has been one of the companies that are utilizing the popularity of mobile usage by encouraging websites to use their Google Search Console, the Mobile-Friendly Test, which allows companies to measure up their website to the search engine results and determine how user-friendly their websites are. The closer the keywords are together their ranking will improve based on key terms.^[45]

SEO may generate an adequate return on investment. However, search engines are not paid for organic search traffic, their algorithms change, and there are no guarantees of continued referrals. Due to this lack of guarantee and uncertainty, a business that relies heavily on search engine traffic can suffer major losses if the search engines stop sending visitors.^[61] Search engines can change their algorithms, impacting a website's search engine ranking, possibly resulting in a serious loss of traffic. According to Google's CEO, Eric Schmidt, in 2010, Google made over 500 algorithm changes – almost 1.5 per day.^[62] It is considered a wise business practice for website operators to liberate themselves from dependence on search engine traffic.^[63] In addition to accessibility in terms of web crawlers (addressed above), user web accessibility has become increasingly important for SEO.

Optimization techniques are highly tuned to the dominant search engines in the target market. The search engines' market shares vary from market to market, as does competition. In 2003, Danny Sullivan stated that Google represented about 75% of all searches.^[64] In markets outside the United States, Google's share is often larger, and data showed Google was the dominant search engine worldwide as of 2007.^[65] As of 2006, Google had an 85–90% market share in Germany.^[66] While there were hundreds of SEO firms in the US at that time, there were only about five in Germany.^[66] As of March 2024, Google still had a significant market share of 89.85% in Germany.^[67] As of June 2008, the market share of Google in the UK was close to 90% according to Hitwise.^{[68][obsolete source]} As of March 2024, Google's market share in the UK was 93.61%.^[69]

Successful search engine optimization (SEO) for international markets requires more than just translating web pages. It may also involve registering a domain name with a country-code top-level domain (ccTLD) or a relevant top-level domain (TLD) for the target market, choosing web hosting with a local IP address or server, and using a Content Delivery Network (CDN) to improve website speed and performance globally. It is also important to understand the local culture so that the content feels relevant to the audience. This includes conducting keyword research for each market, using hreflang tags to target the right languages, and building local backlinks. However, the core SEO principles—such as creating high-quality content, improving user experience, and building links—remain the same, regardless of language or region.^[66]

Regional search engines have a strong presence in specific markets:

• China: Baidu leads the market, controlling about 70 to 80% market share.^[70]
• South Korea: Since the end of 2021, Naver, a domestic web portal, has gained prominence in the country.^[71][72]
• Russia: Yandex is the leading search engine in Russia. As of December 2023, it accounted for at least 63.8% of the market share.^[73]

By the early 2000s, businesses recognized that the web and search engines could help them reach global audiences. As a result, the need for multilingual SEO emerged.^[74] In the early years of international SEO development, simple translation was seen as sufficient. However, over time, it became clear that localization and transcreation—adapting content to local language, culture, and emotional resonance—were far more effective than basic translation.^[75]

On October 17, 2002, SearchKing filed suit in the United States District Court, Western District of Oklahoma, against the search engine Google. SearchKing's claim was that Google's tactics to prevent spamdexing constituted a tortious interference with contractual relations. On May 27, 2003, the court granted Google's motion to dismiss the complaint because SearchKing "failed to state a claim upon which relief may be granted."^[76][77]

In March 2006, KinderStart filed a lawsuit against Google over search engine rankings. KinderStart's website was removed from Google's index prior to the lawsuit, and the amount of traffic to the site dropped by 70%. On March 16, 2007, the United States District Court for the Northern District of California (San Jose Division) dismissed KinderStart's complaint without leave to amend and partially granted Google's motion for Rule 11 sanctions against KinderStart's attorney, requiring him to pay part of Google's legal expenses.^[78][79]

• Webmaster Guidelines from Google
• Google Search Quality Evaluators Guidelines (PDF)
• Webmaster resources from Yahoo!
• Webmaster Guidelines from Microsoft Bing
• The Dirty Little Secrets of Search in The New York Times (February 12, 2011)

 

Crawlers usually perform some type of URL normalization in order to avoid crawling the same resource more than once. The term URL normalization, also called URL canonicalization, refers to the process of modifying and standardizing a URL in a consistent manner. There are several types of normalization that may be performed including conversion of URLs to lowercase, removal of "." and ".." segments, and adding trailing slashes to the non-empty path component.^[18]

Some crawlers intend to download/upload as many resources as possible from a particular web site. So path-ascending crawler was introduced that would ascend to every path in each URL that it intends to crawl.^[19] For example, when given a seed URL of http://llama.org/hamster/monkey/page.html, it will attempt to crawl /hamster/monkey/, /hamster/, and /. Cothey found that a path-ascending crawler was very effective in finding isolated resources, or resources for which no inbound link would have been found in regular crawling.

The importance of a page for a crawler can also be expressed as a function of the similarity of a page to a given query. Web crawlers that attempt to download pages that are similar to each other are called focused crawler or topical crawlers. The concepts of topical and focused crawling were first introduced by Filippo Menczer^[20][21] and by Soumen Chakrabarti et al.^[22]

The main problem in focused crawling is that in the context of a Web crawler, we would like to be able to predict the similarity of the text of a given page to the query before actually downloading the page. A possible predictor is the anchor text of links; this was the approach taken by Pinkerton^[23] in the first web crawler of the early days of the Web. Diligenti et al.^[24] propose using the complete content of the pages already visited to infer the similarity between the driving query and the pages that have not been visited yet. The performance of a focused crawling depends mostly on the richness of links in the specific topic being searched, and a focused crawling usually relies on a general Web search engine for providing starting points.

An example of the focused crawlers are academic crawlers, which crawls free-access academic related documents, such as the citeseerxbot, which is the crawler of CiteSeer^X search engine. Other academic search engines are Google Scholar and Microsoft Academic Search etc. Because most academic papers are published in PDF formats, such kind of crawler is particularly interested in crawling PDF, PostScript files, Microsoft Word including their zipped formats. Because of this, general open-source crawlers, such as Heritrix, must be customized to filter out other MIME types, or a middleware is used to extract these documents out and import them to the focused crawl database and repository.^[25] Identifying whether these documents are academic or not is challenging and can add a significant overhead to the crawling process, so this is performed as a post crawling process using machine learning or regular expression algorithms. These academic documents are usually obtained from home pages of faculties and students or from publication page of research institutes. Because academic documents make up only a small fraction of all web pages, a good seed selection is important in boosting the efficiencies of these web crawlers.^[26] Other academic crawlers may download plain text and HTML files, that contains metadata of academic papers, such as titles, papers, and abstracts. This increases the overall number of papers, but a significant fraction may not provide free PDF downloads.

Another type of focused crawlers is semantic focused crawler, which makes use of domain ontologies to represent topical maps and link Web pages with relevant ontological concepts for the selection and categorization purposes.^[27] In addition, ontologies can be automatically updated in the crawling process. Dong et al.^[28] introduced such an ontology-learning-based crawler using a support-vector machine to update the content of ontological concepts when crawling Web pages.

The Web has a very dynamic nature, and crawling a fraction of the Web can take weeks or months. By the time a Web crawler has finished its crawl, many events could have happened, including creations, updates, and deletions.

From the search engine's point of view, there is a cost associated with not detecting an event, and thus having an outdated copy of a resource. The most-used cost functions are freshness and age.^[29]

Freshness: This is a binary measure that indicates whether the local copy is accurate or not. The freshness of a page p in the repository at time t is defined as:

$\displaystyle F_p(t)=\begincases1&\rm if~p~\rm ~is~equal~to~the~local~copy~at~time~t\\0&\rm otherwise\endcases$

Age: This is a measure that indicates how outdated the local copy is. The age of a page p in the repository, at time t is defined as:

$\displaystyle A_p(t)=\begincases0&\rm if~p~\rm ~is~not~modified~at~time~t\\t-\rm modification~time~of~p&\rm otherwise\endcases$

Coffman et al. worked with a definition of the objective of a Web crawler that is equivalent to freshness, but use a different wording: they propose that a crawler must minimize the fraction of time pages remain outdated. They also noted that the problem of Web crawling can be modeled as a multiple-queue, single-server polling system, on which the Web crawler is the server and the Web sites are the queues. Page modifications are the arrival of the customers, and switch-over times are the interval between page accesses to a single Web site. Under this model, mean waiting time for a customer in the polling system is equivalent to the average age for the Web crawler.^[30]

The objective of the crawler is to keep the average freshness of pages in its collection as high as possible, or to keep the average age of pages as low as possible. These objectives are not equivalent: in the first case, the crawler is just concerned with how many pages are outdated, while in the second case, the crawler is concerned with how old the local copies of pages are.

Two simple re-visiting policies were studied by Cho and Garcia-Molina:^[31]

• Uniform policy: This involves re-visiting all pages in the collection with the same frequency, regardless of their rates of change.
• Proportional policy: This involves re-visiting more often the pages that change more frequently. The visiting frequency is directly proportional to the (estimated) change frequency.

In both cases, the repeated crawling order of pages can be done either in a random or a fixed order.

Cho and Garcia-Molina proved the surprising result that, in terms of average freshness, the uniform policy outperforms the proportional policy in both a simulated Web and a real Web crawl. Intuitively, the reasoning is that, as web crawlers have a limit to how many pages they can crawl in a given time frame, (1) they will allocate too many new crawls to rapidly changing pages at the expense of less frequently updating pages, and (2) the freshness of rapidly changing pages lasts for shorter period than that of less frequently changing pages. In other words, a proportional policy allocates more resources to crawling frequently updating pages, but experiences less overall freshness time from them.

To improve freshness, the crawler should penalize the elements that change too often.^[32] The optimal re-visiting policy is neither the uniform policy nor the proportional policy. The optimal method for keeping average freshness high includes ignoring the pages that change too often, and the optimal for keeping average age low is to use access frequencies that monotonically (and sub-linearly) increase with the rate of change of each page. In both cases, the optimal is closer to the uniform policy than to the proportional policy: as Coffman et al. note, "in order to minimize the expected obsolescence time, the accesses to any particular page should be kept as evenly spaced as possible".^[30] Explicit formulas for the re-visit policy are not attainable in general, but they are obtained numerically, as they depend on the distribution of page changes. Cho and Garcia-Molina show that the exponential distribution is a good fit for describing page changes,^[32] while Ipeirotis et al. show how to use statistical tools to discover parameters that affect this distribution.^[33] The re-visiting policies considered here regard all pages as homogeneous in terms of quality ("all pages on the Web are worth the same"), something that is not a realistic scenario, so further information about the Web page quality should be included to achieve a better crawling policy.

Crawlers can retrieve data much quicker and in greater depth than human searchers, so they can have a crippling impact on the performance of a site. If a single crawler is performing multiple requests per second and/or downloading large files, a server can have a hard time keeping up with requests from multiple crawlers.

As noted by Koster, the use of Web crawlers is useful for a number of tasks, but comes with a price for the general community.^[34] The costs of using Web crawlers include:

• network resources, as crawlers require considerable bandwidth and operate with a high degree of parallelism during a long period of time;
• server overload, especially if the frequency of accesses to a given server is too high;
• poorly written crawlers, which can crash servers or routers, or which download pages they cannot handle; and
• personal crawlers that, if deployed by too many users, can disrupt networks and Web servers.

A partial solution to these problems is the robots exclusion protocol, also known as the robots.txt protocol that is a standard for administrators to indicate which parts of their Web servers should not be accessed by crawlers.^[35] This standard does not include a suggestion for the interval of visits to the same server, even though this interval is the most effective way of avoiding server overload. Recently commercial search engines like Google, Ask Jeeves, MSN and Yahoo! Search are able to use an extra "Crawl-delay:" parameter in the robots.txt file to indicate the number of seconds to delay between requests.

The first proposed interval between successive pageloads was 60 seconds.^[36] However, if pages were downloaded at this rate from a website with more than 100,000 pages over a perfect connection with zero latency and infinite bandwidth, it would take more than 2 months to download only that entire Web site; also, only a fraction of the resources from that Web server would be used.

Cho uses 10 seconds as an interval for accesses,^[31] and the WIRE crawler uses 15 seconds as the default.^[37] The MercatorWeb crawler follows an adaptive politeness policy: if it took t seconds to download a document from a given server, the crawler waits for 10t seconds before downloading the next page.^[38] Dill et al. use 1 second.^[39]

For those using Web crawlers for research purposes, a more detailed cost-benefit analysis is needed and ethical considerations should be taken into account when deciding where to crawl and how fast to crawl.^[40]

Anecdotal evidence from access logs shows that access intervals from known crawlers vary between 20 seconds and 3–4 minutes. It is worth noticing that even when being very polite, and taking all the safeguards to avoid overloading Web servers, some complaints from Web server administrators are received. Sergey Brin and Larry Page noted in 1998, "... running a crawler which connects to more than half a million servers ... generates a fair amount of e-mail and phone calls. Because of the vast number of people coming on line, there are always those who do not know what a crawler is, because this is the first one they have seen."^[41]

A parallel crawler is a crawler that runs multiple processes in parallel. The goal is to maximize the download rate while minimizing the overhead from parallelization and to avoid repeated downloads of the same page. To avoid downloading the same page more than once, the crawling system requires a policy for assigning the new URLs discovered during the crawling process, as the same URL can be found by two different crawling processes.

A crawler must not only have a good crawling strategy, as noted in the previous sections, but it should also have a highly optimized architecture.

Shkapenyuk and Suel noted that:^[42]

While it is fairly easy to build a slow crawler that downloads a few pages per second for a short period of time, building a high-performance system that can download hundreds of millions of pages over several weeks presents a number of challenges in system design, I/O and network efficiency, and robustness and manageability.

Web crawlers are a central part of search engines, and details on their algorithms and architecture are kept as business secrets. When crawler designs are published, there is often an important lack of detail that prevents others from reproducing the work. There are also emerging concerns about "search engine spamming", which prevent major search engines from publishing their ranking algorithms.

While most of the website owners are keen to have their pages indexed as broadly as possible to have strong presence in search engines, web crawling can also have unintended consequences and lead to a compromise or data breach if a search engine indexes resources that should not be publicly available, or pages revealing potentially vulnerable versions of software.

Apart from standard web application security recommendations website owners can reduce their exposure to opportunistic hacking by only allowing search engines to index the public parts of their websites (with robots.txt) and explicitly blocking them from indexing transactional parts (login pages, private pages, etc.).

Web crawlers typically identify themselves to a Web server by using the User-agent field of an HTTP request. Web site administrators typically examine their Web servers' log and use the user agent field to determine which crawlers have visited the web server and how often. The user agent field may include a URL where the Web site administrator may find out more information about the crawler. Examining Web server log is tedious task, and therefore some administrators use tools to identify, track and verify Web crawlers. Spambots and other malicious Web crawlers are unlikely to place identifying information in the user agent field, or they may mask their identity as a browser or other well-known crawler.

Web site administrators prefer Web crawlers to identify themselves so that they can contact the owner if needed. In some cases, crawlers may be accidentally trapped in a crawler trap or they may be overloading a Web server with requests, and the owner needs to stop the crawler. Identification is also useful for administrators that are interested in knowing when they may expect their Web pages to be indexed by a particular search engine.

A vast amount of web pages lie in the deep or invisible web.^[43] These pages are typically only accessible by submitting queries to a database, and regular crawlers are unable to find these pages if there are no links that point to them. Google's Sitemaps protocol and mod oai^[44] are intended to allow discovery of these deep-Web resources.

Deep web crawling also multiplies the number of web links to be crawled. Some crawlers only take some of the URLs in <a href="URL"> form. In some cases, such as the Googlebot, Web crawling is done on all text contained inside the hypertext content, tags, or text.

Strategic approaches may be taken to target deep Web content. With a technique called screen scraping, specialized software may be customized to automatically and repeatedly query a given Web form with the intention of aggregating the resulting data. Such software can be used to span multiple Web forms across multiple Websites. Data extracted from the results of one Web form submission can be taken and applied as input to another Web form thus establishing continuity across the Deep Web in a way not possible with traditional web crawlers.^[45]

Pages built on AJAX are among those causing problems to web crawlers. Google has proposed a format of AJAX calls that their bot can recognize and index.^[46]

There are a number of "visual web scraper/crawler" products available on the web which will crawl pages and structure data into columns and rows based on the users requirements. One of the main difference between a classic and a visual crawler is the level of programming ability required to set up a crawler. The latest generation of "visual scrapers" remove the majority of the programming skill needed to be able to program and start a crawl to scrape web data.

The visual scraping/crawling method relies on the user "teaching" a piece of crawler technology, which then follows patterns in semi-structured data sources. The dominant method for teaching a visual crawler is by highlighting data in a browser and training columns and rows. While the technology is not new, for example it was the basis of Needlebase which has been bought by Google (as part of a larger acquisition of ITA Labs^[47]), there is continued growth and investment in this area by investors and end-users.^{[citation needed]}

The following is a list of published crawler architectures for general-purpose crawlers (excluding focused web crawlers), with a brief description that includes the names given to the different components and outstanding features:

• WolfBot was a massively multi threaded crawler built in 2001 by Mani Singh a Civil Engineering graduate from the University of California at Davis.
• World Wide Web Worm was a crawler used to build a simple index of document titles and URLs. The index could be searched by using the grep Unix command.
• Yahoo! Slurp was the name of the Yahoo! Search crawler until Yahoo! contracted with Microsoft to use Bingbot instead.

• Applebot is Apple's web crawler. It supports Siri and other products.^[48]
• Bingbot is the name of Microsoft's Bing webcrawler. It replaced Msnbot.
• Baiduspider is Baidu's web crawler.
• DuckDuckBot is DuckDuckGo's web crawler.
• Googlebot is described in some detail, but the reference is only about an early version of its architecture, which was written in C++ and Python. The crawler was integrated with the indexing process, because text parsing was done for full-text indexing and also for URL extraction. There is a URL server that sends lists of URLs to be fetched by several crawling processes. During parsing, the URLs found were passed to a URL server that checked if the URL have been previously seen. If not, the URL was added to the queue of the URL server.
• WebCrawler was used to build the first publicly available full-text index of a subset of the Web. It was based on lib-WWW to download pages, and another program to parse and order URLs for breadth-first exploration of the Web graph. It also included a real-time crawler that followed links based on the similarity of the anchor text with the provided query.
• WebFountain is a distributed, modular crawler similar to Mercator but written in C++.
• Xenon is a web crawler used by government tax authorities to detect fraud.^[49][50]

The following web crawlers are available, for a price::

• Diffbot - programmatic general web crawler, available as an API
• SortSite - crawler for analyzing websites, available for Windows and Mac OS
• Swiftbot - Swiftype's web crawler, available as software as a service
• Aleph Search - web crawler allowing massive collection with high scalability

• Apache Nutch is a highly extensible and scalable web crawler written in Java and released under an Apache License. It is based on Apache Hadoop and can be used with Apache Solr or Elasticsearch.
• Grub was an open source distributed search crawler that Wikia Search used to crawl the web.
• Heritrix is the Internet Archive's archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java.
• ht://Dig includes a Web crawler in its indexing engine.
• HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released under the GPL.
• Norconex Web Crawler is a highly extensible Web Crawler written in Java and released under an Apache License. It can be used with many repositories such as Apache Solr, Elasticsearch, Microsoft Azure Cognitive Search, Amazon CloudSearch and more.
• mnoGoSearch is a crawler, indexer and a search engine written in C and licensed under the GPL (*NIX machines only)
• Open Search Server is a search engine and web crawler software release under the GPL.
• Scrapy, an open source webcrawler framework, written in python (licensed under BSD).
• Seeks, a free distributed search engine (licensed under AGPL).
• StormCrawler, a collection of resources for building low-latency, scalable web crawlers on Apache Storm (Apache License).
• tkWWW Robot, a crawler based on the tkWWW web browser (licensed under GPL).
• GNU Wget is a command-line-operated crawler written in C and released under the GPL. It is typically used to mirror Web and FTP sites.
• YaCy, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL).

• Automatic indexing
• Gnutella crawler
• Web archiving
• Webgraph
• Website mirroring software
• Search Engine Scraping
• Web scraping

• Cho, Junghoo, "Web Crawling Project", UCLA Computer Science Department.
• A History of Search Engines, from Wiley
• WIVET is a benchmarking project by OWASP, which aims to measure if a web crawler can identify all the hyperlinks in a target website.
• Shestakov, Denis, "Current Challenges in Web Crawling" and "Intelligent Web Crawling", slides for tutorials given at ICWE'13 and WI-IAT'13.

 

Parramatta has a humid subtropical climate (Köppen climate classification: Cfa) with mild to cool, somewhat short winters and warm to usually hot summers, alongside moderate rainfall spread throughout the year.

Summer maximum temperatures are quite variable, often reaching above 35 °C (95 °F), on average 13.1 days in the summer season, and sometimes remaining in the low 20s, especially after a cold front or a sea breeze, such as the southerly buster. Northwesterlies can occasionally bring hot winds from the desert that can raise temperatures higher than 40 °C (104 °F) mostly from November to February, and sometimes above 44 °C (111 °F) in January severe heatwaves. The record highest temperature (since 1967) was 47.0 °C (116.6 °F) on 4 January 2020. Parramatta is warmer than Sydney CBD in the summer due to the urban heat island effect and its inland location. In extreme cases though, it can be 5–10 °C (9–18 °F) warmer than Sydney, especially when sea breezes do not penetrate inland on hot summer and spring days. For example, on 28 November 2009, the city reached 29.3 °C (84.7 °F),^[24] while Parramatta reached 39.0 °C (102.2 °F),^[25] almost 10 °C (18 °F) higher. In the summer, Parramatta, among other places in western Sydney, can often be the hottest place in the world because of the Blue Mountains trapping hot air in the region, in addition to the UHI effect.^[26]

Rainfall is slightly higher during the first three months of the year because the anticlockwise-rotating subtropical high is to the south of the country, thereby allowing moist easterlies from the Tasman Sea to penetrate the city.^[27][28] The second half of the year tends to be drier (late winter/spring) since the subtropical high is to the north of the city, thus permitting dry westerlies from the interior to dominate.^[29] Drier winters are also owed to its position on the leeward side of the Great Dividing Range, which block westerly cold fronts (that are more common in late winter) and thus would become foehn winds, whereby allowing decent amount of sunny days and relatively low precipitation in that period.^[30] Thunderstorms are common in the months from early spring to early autumn, occasionally quite severe thunderstorms can occur. Snow is virtually unknown, having been recorded only in 1836 and 1896^[31] Parrammatta gets 106.6 days of clear skies annually.

Depending on the wind direction, summer weather may be humid or dry, though the humidity is mostly in the comfortable range, with the late summer/autumn period having a higher average humidity than late winter/early spring.

Church Street is home to many shops and restaurants. The northern end of Church Street, close to Lennox Bridge, features al fresco dining with a diverse range of cuisines. Immediately south of the CBD Church Street is known across Sydney as 'Auto Alley' for the many car dealerships lining both sides of the street as far as the M4 Motorway.^[33]

Since 2000, Parramatta has seen the consolidation of its role as a government centre, with the relocation of agencies such as the New South Wales Police Force Headquarters and the Sydney Water Corporation^[12] from Sydney CBD. At the same time, major construction work occurred around the railway station with the expansion of Westfield Shoppingtown and the creation of a new transport interchange. The western part of the Parramatta CBD is known as the Parramatta Justice Precinct and houses the corporate headquarters of the Department of Communities and Justice. Other legal offices include the Children's Court of New South Wales and the Sydney West Trial Courts, Legal Aid Commission of NSW, Office of Trustee and Guardian (formerly the Office of the Protective Commissioner), NSW Registry of Births, Deaths and Marriages, and the Office of the Director of Public Prosecutions. Nearby on Marsden Street is the Parramatta Courthouse and the Drug Court of New South Wales. The Garfield Barwick Commonwealth Law Courts Building (named in honour of Sir Garfield Barwick), houses courts of the Federal Magistrates Court and the Family Court of Australia. The NSW Government has also announced plans to secure up to 45,000 m² of new A-grade leased office space in Parramatta to relocate a further 4,000 workers from the Sydney CBD.^[34]

Parramatta Square (previously known as Civic Place) is a civic precinct located in the heart of the city, adjacent to Parramatta Town Hall. The Parramatta Square construction works included a redevelopment of the Parramatta Civic Centre, construction of a new culture and arts centre, and the construction of a new plaza. The designs of the first two projects, a 65-storey residential skyscraper and an office building were announced on 20 July 2012.^[35] Concerns from CASA about infringements into controlled airspace from the height of the residential tower resulted in 8 Parramatta Square being turned into a 55-story commercial building, rather than the originally proposed 65-storey residential tower.^[36] Parramatta Square became home to 3,000 National Australia Bank employees, relocated from the Sydney CBD.^[37] Other notable commercial tenants who have established a presence at Parramatta Square include Westpac, Endeavour Energy, KPMG and Deloitte.^[38]

Centenary Square, formerly known as Centenary Plaza, was created in 1975 when the then Parramatta City Council closed a section of the main street to traffic to create a pedestrian plaza. It features an 1888 Centennial Memorial Fountain and adjoins the 1883 Parramatta Town Hall and St John's Cathedral.^[39]

A hospital known as The Colonial Hospital was established in Parramatta in 1818.^[40] This then became Parramatta District Hospital. Jeffery House was built in the 1940s. With the construction of the nearby Westmead Hospital complex public hospital services in Parramatta were reduced but after refurbishment Jeffery House again provides clinical health services. Nearby, Brislington House has had a long history with health services. It is the oldest colonial building in Parramatta, dating to 1821.^[41] It became a doctors residence before being incorporated into the Parramatta Hospital in 1949.

Parramatta is a major business and commercial centre, and home to Westfield Parramatta, the tenth largest shopping centre in Australia.^[42] Parramatta is also the major transport hub for Western Sydney, servicing trains and buses, as well as having a ferry wharf and future light rail and metro services. Major upgrades have occurred around Parramatta railway station with the creation of a new transport interchange, and the ongoing development of the Parramatta Square local government precinct.^[43]

Church Street takes its name from St John's Cathedral (Anglican), which was built in 1802 and is the oldest church in Parramatta. While the present building is not the first on the site, the towers were built during the time of Governor Macquarie, and were based on those of the church at Reculver, England, at the suggestion of his wife, Elizabeth.^[44] The historic St John's Cemetery is located nearby on O'Connell Street.^[45]

St Patrick's Cathedral (Roman Catholic) is one of the oldest Catholic churches in Australia. Construction commenced in 1836, but it wasn't officially complete until 1837. In 1854 a new church was commissioned, although the tower was not completed until 1880, with the spire following in 1883.^[46] It was built on the site to meet the needs of a growing congregation. It was destroyed by fire in 1996, with only the stone walls remaining.

On 29 November 2003, the new St Patrick's Cathedral was dedicated.^[47] The historic St Patrick's Cemetery is located in North Parramatta. The Uniting Church is represented by Leigh Memorial Church.^[48] Parramatta Salvation Army is one of the oldest active Salvation Army Corps in Australia. Parramatta is also home to the Parramatta and Districts Synagogue, which services the Jewish community of western Sydney.^[49]

The Greek Orthodox Parish and Community of St Ioannis (St John The Frontrunner) Greek Orthodox Church was established in Parramatta in May 1960 under the ecumenical jurisdiction of the Greek Orthodox Archdiocese of Australia to serve the predominantly emigrating Greek population of Greater Western Sydney. Originally, the liturgies were held in the hall of St John's Ambulance Brigade in Harris Park until the completion of the church in December 1966 located in Hassall Street Parramatta. The parish sold this property in 2014 and is now located at the corner of George and Purchase Streets.^[50] The Parish Community of St Ioannis continues to serve over 5,000 Greek parishioners.^[51]

A Buddhist temple is located in Cowper Street, Parramatta.^[52] Parramatta's Mosque is in an apartment building on Marsden Street, Parramatta.^[53] The district is served by BAPS Swaminarayan Hindu temple located on Eleanor St, Rosehill,^[54] and a Murugan Hindu temple in Mays Hill, off Great Western Highway.^[55]

Parramatta Park is a large park adjacent to Western Sydney Stadium that is a popular venue for walking, jogging and bike riding. It was formerly the Governor's Domain, being land set aside for the Governor to supply his farming needs, until it was gazetted as a public park in 1858.^[56] As the Governor's Domain, the grounds were considerably larger than the current 85 hectare Parramatta Park, extending from Parramatta Road in the south as evident by a small gatehouse adjacent to Parramatta High School. For a time Parramatta Park housed a zoo^[57] until 1951 when the animals were transferred to Taronga Zoo.

Parramatta is known as the 'River City' as the Parramatta River flows through the Parramatta CBD.^[58] Its foreshore features a playground, seating, picnic tables and pathways that are increasingly popular with residents, visitors and CBD workers.^[59]

Prince Alfred Square is a Victorian era park located within the CBD on the northern side of the Parramatta River. It is one of the oldest public parks in New South Wales with trees dating from c. 1869. Prior to being a public park, it was the site of Parramatta's second gaol from 1804 until 1841 and the first female factory in Australia between 1804 and 1821.

In contrast to the high level of car dependency throughout Sydney, a greater proportion of Parramatta's workers travelled to work on public transport (45.2%) than by car (36.2%) in 2016.^[60]

Parramatta railway station is served by Sydney Trains' Cumberland Line, Leppington & Inner West Line and North Shore & Western Line services.^[61] NSW TrainLink operates intercity services on the Blue Mountains Line as well as services to rural New South Wales. The station was originally opened in 1855, located in what is now Granville, and known as Parramatta Junction. The station was moved to its current location and opened on 4 July 1860, five years after the first railway line in Sydney was opened, running from Sydney to Parramatta Junction.^[62] It was upgraded in the 2000s, with work beginning in late 2003 and the new interchange opening on 19 February 2006.^[63]

The light rail Westmead & Carlingford Line runs from Westmead to Carlingford via the Parramatta city centre. A future branch will run to Sydney Olympic Park.^[64]

The under construction Sydney Metro West will be a metro line run between the Sydney central business district and Westmead. Announced in 2016,^[65] the line is set to open in 2032 with a station in Parramatta.^[66]

Parramatta is also serviced by a major bus interchange located on the south eastern side of the railway station. The interchange is served by buses utilising the North-West T-way to Rouse Hill and the Liverpool–Parramatta T-way to Liverpool. Parramatta is also serviced by one high frequency Metrobus service:

• M91 – Parramatta to Hurstville via Granville, Bankstown and Peakhurst

A free bus Route 900 is operated by Transit Systems in conjunction with the state government. Route 900 circles Parramatta CBD.^[67] A free bus also links Western Sydney Stadium to Parramatta railway station during major sporting events.

The Parramatta ferry wharf is at the Charles Street Weir, which divides the tidal saltwater from the freshwater of the upper river, on the eastern boundary of the Central Business District. The wharf is the westernmost destination of Sydney Ferries' Parramatta River ferry services.^[68]

Parramatta Road has always been an important thoroughfare for Sydney from its earliest days. From Parramatta the major western road for the state is the Great Western Highway. The M4 Western Motorway, running parallel to the Great Western Highway has taken much of the traffic away from these roads, with entrance and exit ramps close to Parramatta.

James Ruse Drive serves as a partial ring-road circling around the eastern part of Parramatta to join with the Cumberland Highway to the north west of the city.

The main north-south route through Parramatta is Church Street. To the north it becomes Windsor Road, and to the south it becomes Woodville Road.

Historical population

Year	Pop.	±%
2001	17,982	—
2006	18,448	+2.6%
2011	19,745	+7.0%
2016	25,798	+30.7%
2021	30,211	+17.1%

According to the 2016 census conducted by the Australian Bureau of Statistics, the suburb of Parramatta had a population of 30,211. Of these:^[69]

Ethnic diversity

The most common country of birth in Parramatta is India representing 30.9% of the population, outnumbering Australian born residents at 24.8%. The next most common are China 8.9%, Nepal 5.5%, Philippines 2.5% and Iran 1.3%. However, only 6.8% identify their ancestry as Australian; the other common self-identified ancestries were Indian 27.3%, Chinese 15.3%, English 8.5% and Nepali 5.5%. About one quarter (24.4%) of people spoke English at home; other languages spoken at home included Hindi 10.4%, Mandarin 8.8%, Nepali 5.3%, Tamil 5.0% and Telugu 4.3%.

Religion

This question is optional in the Census. Of the people who answered it, the most common response was Hinduism 33.6%; the next most common responses were "No Religion" 21.6%, Catholic 12.1%, Not stated 7.7% and Islam 7.5%.

Age distribution

Parramatta has an over-representation of young adults when compared to the country as a whole. Parramatta residents' median age was 32 years, compared to the national median of 38. Children aged under 15 years made up 16.3% of the population (national average is 18.2%) and people aged 65 years and over made up 6.6% of the population (national average is 17.2%).

Income

The average weekly household income was $2,092, compared to the national average of $1,746.

Housing

The majority of dwellings in Parramatta (85.6%) were flats, units or apartments; 7.7% were separate houses, and 5.7% were semi-detached (mostly townhouses). The average household size was 2.4 people. In 2021, 2.2% of households were public housing, compared to 6.3% in 2016.^[70]

• Keith Agget (1931–2017), rugby league player
• Bernie Banton (1946–2007), builder and social justice campaigner
• Richie Benaud (1930–2015), cricketer and commentator
• Allan Cunningham (1791–1839), explorer and botanist
• Greg Dyer (born 1959), cricketer
• Gerry Hazlitt (1888–1915), cricketer
• Paul Hogan (born 1939), comedian and actor
• Harry Hopman (1906–1985), tennis player
• David Lennox (1788–1873), colonial bridge builder
• John Lewin (1770–1819), first professional artist in New South Wales
• Bruce Mann (1926–2007), rugby league player
• George McIver (1859–1945), science fiction writer
• Rev. Samuel Marsden (1765–1838), known as the "flogging parson"
• Mary Cover Hassall (1799–1825), Methodist missionary to Tonga Island
• Dowell Philip O'Reilly (1865–1923), poet and politician
• Todd Payten (born 1979), rugby league player and coach
• "Jock" Ross (born 1943), outlaw biker.
• Nora Kate Weston (1880-1965), artisan
• J. C. Wharton (1853–1929), editor of Parramatta Times (defunct) and a local history^[71]

Parramatta is home to several primary and secondary schools. Arthur Phillip High School was established in 1960 in its own right, in buildings which had been used continuously as a school since 1875 is the oldest continuously operating public school in Parramatta. Parramatta High School was the first coeducational school in the Sydney metropolitan area established in 1913. Our Lady of Mercy College is one of the oldest Catholic schools in Australia. Macarthur Girls High School is successor to an earlier school 'Parramatta Commercial and Household Arts School'. Others schools include Parramatta Public School, Parramatta East Public School, Parramatta West Public School, and St Patrick's Primary Parramatta.

Several tertiary education facilities are also located within Parramatta. A University of New England study centre and two Western Sydney University campuses are situated in Parramatta. The Western Sydney University Parramatta Campus consists of two sites: Parramatta South (the primary site) which occupies the site of the historic Female Orphan School^[72] and Parramatta North (the secondary site) which includes the adjacent Western Sydney University Village Parramatta (formerly UWS Village Parramatta) an on campus student village accommodation. Whereby, the flagship Parramatta City Campus Precinct consists of two buildings: the Engineering Innovation Hub located at 6 Hassall Street and the Peter Shergold Building located at 1 Parramatta Square (169 Macquarie Street).^[73] Alphacrucis University College is a Christian liberal arts college with a campus in Parramatta located at 30 Cowper Street.^[74] The University of Sydney has also announced that it intends to establish a new campus in Parramatta.^[75]

The Parramatta Advertiser is the local newspaper serving Parramatta and surrounding suburbs.

On 16 March 2020, the Australian Broadcasting Corporation opened a new Western Sydney newsroom in Horwood Place at Parramatta incorporating space for 12 staff and news production equipment with the capacity to broadcast live radio programs.^[76] According to the ABC, the opening formed part of its strategic goal to improve its presence in outer metropolitan areas.^[76] Additionally, the ABC announced on 16 June 2021 its intention to relocate approximately 300 employees to Parramatta, which is part of a five-year plan which aims to have 75% of its content makers based away from the network's Ultimo headquarters by 2025.^[77][78]

As the centre of the City of Parramatta, as well as the centre and second largest business district of Sydney, Parramatta hosts many festivals and events.^[79] Riverside Theatres is a performing arts centre located on the northern bank of Parramatta River. The city hosts the following events:

• January – Sydney Festival and Australia Day^[80]
• February – Lunar New Year and Tropfest^[81]
• April – Anzac Day
• July – Winterlight and Burramatta Day (Naidoc)
• October – Parramasala and Parramatta Lanes^[82]
• November – Loy Krathong, Christmas in Parramatta and Foundation Day
• December – New Year's Eve

Parramatta Park contains Old Government House and thus Parramatta was once the capital of the colony of New South Wales until Governors returned to residing in Sydney in 1846.^[83] Another feature is the natural amphitheatre located on one of the bends of the river, named by Governor Philip as "the Crescent", which is used to stage concerts. It is home to the Dairy Cottage, built from 1798 to 1805, originally a single-room cottage and is one of the earliest surviving cottages in Australia.

The remains of Governor Brisbane's private astronomical observatory, constructed in 1822, are visible. Astronomers who worked at the observatory, discovering thousands of new stars and deep sky objects, include James Dunlop and Carl Rümker. In 1822, the architect S. L. Harris designed the Bath House for Governor Brisbane and built it in 1823. Water was pumped to the building through lead pipes from the river. In 1886, it was converted into a pavilion.^[84]

• The Rosehill Race Course holds various race meets throughout the year, including: Derby Day, Golden Rose Day, and Rosehill Gardens Race Day.
• The Parramatta Farmers Markets^[85] occurs every Friday, and has local produce.

Parramatta is the home of several professional sports teams. These teams include the Parramatta Eels of the National Rugby League and Western Sydney Wanderers of the A-League. Both teams formerly played matches at Parramatta Stadium that has since been demolished, and replaced with the 30,000-seat Western Sydney Stadium.^[86] Parramatta Stadium was also home to the now dissolved Sydney Wave of the former Australian Baseball League and Parramatta Power of the former National Soccer League. The newly built Bankwest Stadium opened its gates for the community on 14 April 2019 with free entry for all fans. Located on O’Connell Street, the stadium is in proximity of the Parramatta CBD. The opening sporting event was the 2019 Round 6 NRL clash between Western Sydney rivals the Parramatta Eels and Wests Tigers on Easter Monday 22 April. The Eels won the match by a score of 51–6. It is being predicted that the new stadium will boost Western Sydney economy by contributing millions of dollars to it.^[87]

Duran Duran's “Union of the Snake” music video with Russell Mulcahy was filmed in 1983 at Parramatta using 35mm film.^[88]

The 2013 superhero film The Wolverine used the intersection of George Street and Smith Street as a filming location to depict Tokyo, Japan.^[89]

Parramatta has a number of heritage-listed sites, including:

• List of tallest buildings in Parramatta
• Parramatta cloth
• Story Factory

• Parramatta City Council website
• Parramatta Park website
• Parramatta & District Historical Society Inc website

• "Parramatta". Dictionary of Sydney. Retrieved 28 September 2015. [CC-By-SA]
• Chris Levins – Parramatta Park Trust (2010). "Parramatta Park". Dictionary of Sydney. Retrieved 28 September 2015. [CC-By-SA]
• Michaela Ann Cameron (2015). "Brislington". Dictionary of Sydney. Retrieved 2 October 2015. [CC-By-SA]
• Michaela Ann Cameron (2015). "The Crescent". Dictionary of Sydney. Retrieved 6 October 2015. [CC-By-SA]
• Michaela Ann Cameron (2015). "Travellers' Rest Inn Parramatta". Dictionary of Sydney. Dictionary of Sydney Trust. Retrieved 11 October 2015.[CC-By-SA]
• Michaela Ann Cameron (2015). "Roseneath Cottage". Dictionary of Sydney. Dictionary of Sydney Trust. Retrieved 16 October 2015.[CC-By-SA]